PROTOCOL: Assessment of outcome reporting bias in studies included in Campbell systematic reviews.

This is the protocol for a Campbell systematic review. The objectives are as follows: To identify methods used to assess the risk of outcome reporting bias (ORB) in studies included in recent Campbell systematic reviews of intervention effects. The review will answer the following questions: What proportion of recent Campbell reviews included assessment of ORB? How did recent reviews define levels of risk of ORB (what categories, labels, and definitions did they use)? To what extent and how did these reviews use study protocols as sources of data on ORB? To what extent and how did reviews document reasons for judgments about risk of ORB? To what extent and how did reviews assess the inter-rater reliability of ORB ratings? To what extent and how were issues of ORB considered in the review's abstract, plain language summary, and conclusions?

The Campbell Collaboration's systematic review of school-based anti-bullying interventions does not meet mandatory methodological standards.

BACKGROUND: Many published reviews do not meet the widely accepted PRISMA standards for systematic reviews and meta-analysis. Campbell Collaboration and Cochrane reviews are expected to meet even more rigorous standards, but their adherence to these standards is uneven. For example, a newly updated Campbell systematic review of school-based anti-bullying interventions does not appear to meet many of the Campbell Collaboration's mandatory methodological standards. ISSUES: In this commentary, we document methodological problems in the Campbell Collaboration's new school-based anti-bullying interventions review, including (1) unexplained deviations from the protocol; (2) inadequate documentation of search strategies; (3) inconsistent reports on the number of included studies; (4) undocumented risk of bias ratings; (5) assessments of selective outcome reporting bias that are not transparent, not replicable, and appear to systematically underestimate risk of bias; (6) unreliable assessments of risk of publication bias; (7) use of a composite scale that conflates distinct risks of bias; and (8) failure to consider issues related to the strength of the evidence and risks of bias in interpreting results and drawing conclusions. Readers who are unaware of these problems may place more confidence in this review than is warranted. Campbell Collaboration editors declined to publish our comments and declined to issue a public statement of concern about this review. CONCLUSIONS: Systematic reviews are expected to use transparent methods and follow relevant methodological standards. Readers should be concerned when these expectations are not met, because transparency and rigor enhance the trustworthiness of results and conclusions. In the tradition of Donald T. Campbell, there is need for more public debate about the methods and conclusions of systematic reviews, and greater clarity regarding applications of (and adherence to) published standards for systematic reviews.

Ranking of addiction journals in eight widely used impact metrics.

Background and aims: Journal metrics assess impact upon the research literature, and are now used to assess individual researchers in hiring and promotion decisions. This study compared the ranking of addiction journals according to eight widely used metrics; assessed the correlations between journal rankings; and assessed changes over time in metric scores. Methods: Data pertaining to the 2020 scores on eight metrics for 43 journals were obtained and the top 20 ranking in each compared and the correlations between rankings assessed. The Impact Factor was employed to assess changes over time. Results: Ignoring the two categorization systems used by some metrics, 31 journals appeared in at least one metric top 20 and 11 in all eight. The top rank in each was occupied by one of three journals. Three-quarters of the correlations between rankings were above 6.0. The number of journals with an Impact Factor rose from 23 in 1997 to 38 in 2020, and the journals added tended to focus on addictions other than alcohol and drugs or have a specific focus. Conclusions and discussion: The results indicate a concentration of journals at the top of the metrics and moderate to strong agreement between them, but almost three-quarters of journals appeared in at least one metric. The longitudinal data reflect both a broadening and specialization of the addiction field. The study limitations include exclusion of some journals and metrics.

Registration and primary outcome reporting in behavioral health trials.

BACKGROUND: Registration of research studies is designed to lock investigators into a data collection and analysis plan before a study starts and thereby limit their ability to engage in flexible data analysis and selective outcome reporting. Studies of registered clinical trials show that one- to two-thirds are registered after the study has started and that non-adherence to important design and analytic features, such as reporting data pertaining to all primary outcomes, remains high. Less is known about the effects of registration on research transparency and integrity outside of clinical trials. To address this gap in knowledge, the current study examined the effects of registration on the reporting of research findings in a sample of behavioral health trials published in BMC Public Health. METHODS: Registered trials published in the BMC Public Health section "Health Behavior, Health Promotion and Society" between 2011 and 2015 were included in the study. For each trial, we reviewed associated online submissions from 13 different registration sites. For those determined to have been prospectively registered, we used the trial registry, MEDLINE (Pubmed), PsychINFO, Web of Science and e-mails to investigators to identify subsequent publications from the study that reported results pertaining to primary outcomes. The two investigators then independently reviewed the outcome publication(s) and compared the primary outcomes reported in these to the registered primary outcomes. RESULTS: The final analytic sample comprised 136 locatable, registered trials with an identifiable start date. Sixty-eight of the 136 were prospectively registered. Among these prospectively registered trials, only 16 published manuscripts reported outcomes and methods that were concordant with their registrations. CONCLUSIONS: Retrospective submission of protocols for publication and retrospective registration remain common in public health research, and adherence to prespecified outcomes is rare. In its current form, registration of behavioral and health promotion trials is likely to have minimal effect on preventing selective outcome reporting in publications, and the pervasiveness of vague and incomplete registry entries means that registries will have limited utility in terms of facilitating replication studies.

High impact nutrition and dietetics journals' use of publication procedures to increase research transparency.

BACKGROUND: The rigor and integrity of the published research in nutrition studies has come into serious question in recent years. Concerns focus on the use of flexible data analysis practices and selective reporting and the failure of peer review journals to identify and correct these practices. In response, it has been proposed that journals employ editorial procedures designed to improve the transparency of published research. OBJECTIVE: The present study examines the adoption of editorial procedures designed to improve the reporting of empirical studies in the field of nutrition and dietetics research. DESIGN: The instructions for authors of 43 journals included in Quartiles 1 and 2 of the Clarivate Analytics' 2018 Journal Citation Report category Nutrition and Dietetics were reviewed. For journals that published original research, conflict of interest disclosure, recommendation of reporting guidelines, registration of clinical trials, registration of other types of studies, encouraging data sharing, and use of the Registered Reports were assessed. For journals that only published reviews, all of the procedures except clinical trial registration were assessed. RESULTS: Thirty-three journals published original research and 10 published only reviews. Conflict of interest disclosure was required by all 33 original research journals. Use of guidelines, trial registration and encouragement of data sharing were mentioned by 30, 27 and 25 journals, respectively. Registration of other studies was required by eight and none offered Registered Reports as a publication option at the time of the review. All 10 review journals required conflict of interest disclosure, four recommended data sharing and three the use of guidelines. None mentioned the other two procedures. CONCLUSIONS: While nutrition journals have adopted a number of procedures designed to improve the reporting of research findings, their limited effects likely result from the mechanisms through which they influence analytic flexibility and selective reporting and the extent to which they are properly implemented and enforced by journals.

Availability of Research Data in High-Impact Addiction Journals with Data Sharing Policies.

Although data sharing is one of the primary measures proposed to improve the integrity and quality of published research, studies show it remains the exception not the rule. The current study examines the availability of data in papers reporting the results of analyses of empirical data from original research in high-impact addiction journals. Thirteen high-impact journals with data sharing policies were selected from those included in the substance abuse category of the 2018 Clarivate Analytics' Journal Citation Report. The first 10 full or short original research reports that included empirical data in the most recent complete issue of each journal were electronically searched and reviewed for reference to where their data can be obtained and for a formal data sharing statement. Only eight of the 130 papers contained a data sharing statement in their text or supplementary online materials, and just one contained a direct link to the data analyzed. Data sharing was rare in the 13 high-impact addiction journals reviewed. The nature of the data reported in addiction journals might partly explain this. Currently, data sharing is not a procedure likely to improve the quality and integrity of published addiction research.

Use of publication procedures to improve research integrity by addiction journals.

BACKGROUND AND AIMS: The credibility crisis evident in many academic disciplines has led peer-reviewed journals to implement procedures to reduce use of flexible data analysis practices and selective reporting of results. This exploratory study examined the adoption of six of these procedures by addiction journals. METHODS: Thirty-eight high-impact addiction journals were identified using the 2018 Clarivate Analytics Journal Citation Report for 2017 ranks. The online instructions for authors were reviewed for references to six publication procedures: conflict of interest disclosure, reporting guidelines, clinical trial registration, registration of other study designs, data-sharing and registered reports. The webpages of the Center for Open Science and Consolidated Standards of Reporting Trials (CONSORT) were also reviewed for data pertaining to registered reports and reporting guidelines, respectively. RESULTS: The range of procedures adopted by the addiction journals was 0-5, with a mean of 2.66. Conflict-of-interest disclosure was required by all but one journal. Encouraging data-sharing was the next most commonly required procedure. Fewer than half the journals recommended specific reporting guidelines or required registration of clinical trials, and only four required procedures to pre-specify hypotheses and analytical methods. CONCLUSIONS: While many addiction journals have adopted publication procedures to improve research integrity, these can be limited by their voluntary nature and monitoring difficulties. More stringent requirements that lock researchers into specific hypotheses and analyses have not been widely adopted.

A Systems Approach to Understanding and Improving Research Integrity.

Concern about the integrity of empirical research has arisen in recent years in the light of studies showing the vast majority of publications in academic journals report positive results, many of these results are false and cannot be replicated, and many positive results are the product of data dredging and the application of flexible data analysis practices coupled with selective reporting. While a number of potential solutions have been proposed, the effects of these are poorly understood and empirical evaluation of each would take many years. We propose that methods from the systems sciences be used to assess the effects, both positive and negative, of proposed solutions to the problem of declining research integrity such as study registration, Registered Reports, and open access to methods and data. In order to illustrate the potential application of systems science methods to the study of research integrity, we describe three broad types of models: one built on the characteristics of specific academic disciplines; one a diffusion of research norms model conceptualizing researchers as susceptible, "infected" and recovered; and one conceptualizing publications as a product produced by an industry comprised of academics who respond to incentives and disincentives.

Can We Trust Positive Findings of Intervention Research? The Role of Conflict of Interest.

In recent years, there has been increased attention to the issue of conflict of interest within prevention research. The aims of this paper are to discuss these developments and to relate them to discussions of conflict of interest in the broader scientific literature. Although there has been concern expressed about the extent to which conflicts of interest can be defined and measured, empirical research suggests that financial conflicts can be easily identified and assessed in meta-analyses focused on their effects on research quality. Research evidence also shows that conflict of interest is associated with use of flexible data analysis practices and the reporting of chance positive findings, both within prevention research and related disciplines such as public health and psychology. However, the overwhelming majority of published studies report positive results, and there are a number of other influences within academia (such as pressure to publish) that account for this and for the use of flexible data analysis practices. Accordingly, introducing measures to improve research quality in general, rather than just focusing on problems specific to research in which there is a clearly identifiable conflict of interest, may prove more effective and less controversial. Most such efforts focus on introducing greater transparency into research design, practice, and reporting. These both curtail employment of flexible data analysis practices and make their use transparent to investigators seeking to assess their effects on research quality. Also, requiring detailed disclosures of conflicts be reported by all investigators (not just senior authors) would improve current disclosure practices.

Violent crime redistribution in a city following a substantial increase in the number of off-sale alcohol outlets: A Bayesian analysis.

INTRODUCTION AND AIMS: This study examined whether the introduction of a large number of off-premise alcohol outlets into a city over a brief period of time could affect rates of violent crime. DESIGN AND METHODS: The study analysed annual counts of violent crime across 172 US Census block groups in Lubbock, Texas from 2006 through 2011. Spatial Poisson models related annual violent crime counts within each block group to off-premise and on-premise alcohol outlets active during this time period as well as neighbourhood socio-demographic characteristics. The effects of alcohol outlets were assessed both within block groups and across adjacent block groups. RESULTS: On-premise outlets had a small, significant positive association with violence within a given block group. A similar well-supported local effect for off-premise outlets was not found. However, the spatially lagged effect for off-sale premises was well-supported, indicating that greater densities of these outlets were related to greater rates of violent crime in adjacent areas. DISCUSSION AND CONCLUSIONS: While these analyses confirmed a previous time-series analysis in finding no city-wide effect of the increase in off-premise outlets, they do suggest that such outlets in a local area may be related to violence in nearby geographic areas. They indicate the importance of examining neighbourhood-specific effects of alcohol outlets on violence in addition to the city-wide effects. They also present further evidence supporting the need to examine the differential effects of on-sale and off-sale premises.